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Abstract 



The purpose of this paper is to derive optimal rules for variable-length mastery tests in case 
three mastery classification decisions (nonmastery, partial mastery, and mastery) are 
distinguished. In a variable-length or adaptive mastery test, the decision is to classify a subject 
as a master, a partial master, a nonmaster, or continuing sampling and administering another 
test item. The framework of minimax sequential decision theory is used; that is, optimal 
sequential rules minimizing the maximum expected losses associated with all possible 
decision rules. The binomial model is assumed for the conditional probability of a correct 
response given the true level of functioning, whereas threshold loss is adopted for the loss 
function involved. Monotonicity conditions are derived, that is, conditions sufficient for 
optimal sequential rules to be in the form of cutting scores. The paper concludes with an 
empirical example of a computerized adaptive mastery test for concept-learning in medicine. 

Key words: adaptive mastery testing, minimax sequential rules, monotonicity conditions, 

least favorable prior, binomial distribution, threshold loss. 
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Introduction 

Over the past few decades, several adaptive instructional programs have been implemented on 
computers (e.g., Bejar & Braun, 1994; Bork, 1984; De Diana & Vos, 1988; Gegg-Harrison, 
1992; Hambleton, 1974; Hansen, Ross, & Rakow, 1977; Kontos, 1985; Koper 1995; Vos, 
1988, 1994a, 1994b, 1995a, 1995b, 1998a). Although these programs come in many forms, all 
of these forms share the same basic design. All systems are basically series of comparatively 
small instructional units or modules through which students are routed by means of a few 
achievement test items administered right after a module. An important challenge to their 
instructors is to adapt instruction to individual learner differences (aptitudes, prior knowledge) 
and learning needs (amount and sequence of instruction). 

A situation that often arises in such systems is the following: It is desired to classify 
students as either a master or a nonmaster, i.e., those students who pass or fail the mastery test 
at the end of the instructional unit. Students who pass the mastery test may proceed with the 
next, presumably more complex module. Students who fail, however, are retained at the same 
instructional unit. 

The situation described above refers to a fixed-length mastery test, where the 
performance on a fixed number of test items is used for deciding on either mastery or 
nonmastery. The fixed-length mastery problem has been studied extensively in the literature 
within the framework of (empirical) Bayesian decision theory (e.g., Cronbach & Gleser, 1965; 
Davis et al., 1973; De Gruijter & Hambleton, 1984; Hambleton & Novick, 1973; Huynh, 
1976, 1977; Swaminathan et al., 1975; van der Linden, 1980, 1990; van der Linden & 
Mellenbergh, 1977; Wilcox, 1977). In addition, optimal rules for the fixed-length mastery 
problem have also been derived within the framework of the minimax (Wald) strategy (e.g., 
Huynh, 1980; Veldhuijzen, 1982). 

In both approaches, the following two basic elements are distinguished: A 
psychometric model relating observed test scores and student's true level of functioning to 
each other, and a loss structure evaluating the total costs and benefits of all possible decision 
outcomes. Within the framework of Bayesian decision theory, optimal rules (i.e., the Bayes 
rules) are obtained by minimizing the posterior expected loss. The Bayes principle assumes 
that prior knowledge about student's true level of functioning is available and can be 
characterized by a probability distribution called the prior. 
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Using minimax decision theory, optimal rules (i.e., the minimax rules) are obtained by 
minimizing the maximum expected losses associated with all possible decision rules. 
Decision rules are hereby prescriptions specifying for each possible observed test score what 
action has to be taken. In fact, the minimax principle assumes that it is best to prepare for the 
worst and to establish the maximum expected loss for each possible decision rule (e.g., van 
der Linden, 1980). In other words, the minimax decision rule is a bit conservative and 
pessimistic (Coombs, Dawes, & Tversky, 1970). 

Although the minimax principle assumes that no prior knowledge about true level of 
functioning is available, a minimax rule can be conceived as a rule that is based on 
minimization of posterior expected loss as well, but under the restriction that the prior is the 
least favorable element of the class of priors (e.g., Ferguson, 1967, Sect. 1.6). In other words, 
there exists a (least favorable) prior distribution on the true level of functioning such that the 
corresponding Bayes solution is exactly the same as the minimax decision rule (Huynh, 1980). 

The test at the end of the instructional unit does not necessarily have to be a fixed- 
length mastery test but might also be a variable-length mastery test. In this case, in addition to 
the decisions declaring mastery or nonmastery, also the decision of continuing sampling and 
administering another item is available. In fact, variable-length mastery tests are more 
common in individualized instructional programs than fixed-length mastery tests. This is 
because variable-length mastery tests offer the possibility to provide shorter tests for those 
students who have clearly attained a certain level of mastery (or clearly nonmastery) and 
longer tests for whom the mastery decision is not as clear-cut (Lewis & Sheehan, 1990). The 
variable-length mastery test is also known as an adaptive mastery test (AMT). 

The purpose of this paper is to derive optimal decision rules for adaptive mastery 
testing. Doing so, in addition to the decision of continuing sampling and administering 
another item, not two but three possible classification decisions are distinguished, namely 
declaring mastery, partial mastery, or nonmastery. 

A useful application of the derived optimal rules is decision making in computerized 
adaptive instructional systems. The successful implementation of a computerized adaptive 
instructional system depends, at least in part, upon the availability of appropriate testing and 
decision making procedures to guide the student through the system (Hambleton, 1974). For 
instance, if a student is not directed to an appropriate module, his/her motivation may decrease 
due to the instruction not being matched to his/her specific learning characteristics. 
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Several approaches have been proposed for deriving optimal decision rules for the 
variable-length mastery problem. One of the earliest approaches dates back to Ferguson 
(1969a, 1969b) using Wald's sequential probabily ratio test (SPRT). In Ferguson's approach, 
the condional probability of a correct response given the true level of functioning (i.e., the 
psychometric model) is modeled as a binomial distribution. Alternative adaptive mastery 
testing procedures within an SPRT-framework, in combination with item response theory for 
modelling the conditional probability of a correct response given the true level of functioning, 
have been proposed by Kingsbury and Weiss (1983), Reckase (1983), and Spray and Reckase 
(1996). 

Recently, Lewis and Sheehan (1990) and Sheehan and Lewis (1992) have proposed 
Bayesian sequential decision theory (e.g., DeGroot, 1970; Ferguson, 1967; Lehmann, 1959; 
Lindgren, 1976) to derive optimal rules for the variable-length mastery problem. Like in the 
fixed-length mastery problem, optimal sequential rules are obtained by choosing the action 
(i.e., mastery, nonmastery, or continuing sampling) that minimizes posterior expected loss at 
each stage of sampling using techniques of dynamic programming (i.e., backward induction). 
Doing so, the posterior expected loss associated with continuing sampling is determined by 
averaging the posterior expected loss associated with each of the possible future decision 
outcomes relative to the probability of observing those outcomes. For known psychometric 
model, prior, and observed item response vector, this probability can be calculated. As pointed 
out by Lewis and Sheehan (1990), the action chosen at each stage of sampling is optimal with 
respect to the entire adaptive mastery testing procedure. 

In the present paper, the framework of minimax sequential decision theory (e.g., 
DeGroot, 1970; Ferguson, 1967; Lehmann, 1959; Lindgren, 1976) is proposed to derive 
optimal rules for the variable-length mastery problem; that is, minimizing the maximum 
expected losses associated with each of the possible decision outcomes. The maximum 
expected loss associated with continuing sampling is determined again by averaging the 
maximum expected losses associated with each of the possible future decision outcomes 
relative to the probability of observing those outcomes. For the prior needed to compute this 
probability, the least favorable prior will be taken. 

Optimal sequential rules for the adaptive four-action mastery problem, following 
Ferguson (1969a, 1969b), will be derived under the assumption of a binomial model for the 
conditional probability of a correct response given the true level of functioning. The choice of 
^ ^his psychometric model assumes that, given the true level of functioning, each item has the 
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same probability of being correctly answered, or that items are sampled at random. For the 
loss structure involved, the well-known threshold loss function is adopted. Furthermore, 
conditions sufficient for optimal sequential rules to be monotone, that is, in the form of 
cutting scores, are derived. The procedures for sequentially setting optimal cutting scores are 
demonstrated by an empirical example of a computerized adaptive mastery test for concept- 
learning in medicine. 



Notations 

Within the framework of both minimax and Bayesian sequential decision theory, optimal rules 
can be obtained without specifying a maximum test length. In the following, however, an 
adaptive four-action mastery test is supposed to have a maximum length of n (n > 1). As 
pointed out by Ferguson (1969a, 1969b), a maximum test length is needed in order to classify 
within a reasonable period of time those students for whom the decision of declaring mastery, 
partial mastery, or nonmastery is not as clear-cut. 

In the following, the observed item response at each stage of sampling k (1 < k < n) for 
a randomly sampled student will be denoted by a discrete random variable X|^, with 
realization x|^. The observed response variables X|,...,X|^ are assumed to be independent and 
identically distributed for each value of k (1 < k < n), and take the values 0 and 1 for 
respectively correct and incorrect responses to the kth item. Furthermore, let the observed 
number-correct score be denoted by a discrete random variable = X] +...+ X|^ (1 < k < n), 
with realization s|^ = X| +...+ X|^ (0 < S|^ < k). 

Student's true level of functioning is unknown due to measurement and sampling error. 
All that is known is his/her observed number- correct score from a small sample of test items. 
In other words, the mastery test is not a perfect indicator of student's true performance. 
Therefore, let student's true level of functioning be denoted by a continuous random variable 
T, with realization t e [0,1]. 

Finally, assuming X| = X|,...,X|^ = X|^ has been observed, the two basic elements of 
minimax sequential decision making discussed earlier can now be formulated as follows: A 
psychometric model relating observed number-correct score s^ to student’s true level of 
functioning t at each stage of sampling k (1 < k < n), and a loss function describing the loss 
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I(ai(x],...,x|^),t) incurred when action ai(xi,...,x|^) is taken for the student whose true level of 
functioning is t. 



Statement of the Adaptive Four- Action Mastery Problem 

In the following, linking up with common practice in criterion-referenced testing, the decision 
rules for the adaptive four-action mastery problem at each stage of sampling k (1 < k < n) are 
assumed to be in the form of sequential cutting scores Sd(k) and Sc2(k) on the observed 
number-correct scale S^, where Sd(k) < Sc 2 (k). Conditions sufficient for sequentially setting 
cutting scores (i.e., monotonicity conditions) are given later on. 

Given the observed item response vector (x] x|^), the following four actions can be 

taken at each stage of sampling k (1 < k < n): First, the action nonmastery ai(x|,...,xk) is 

taken if sj^ < Sd(k). Second, the action partial mastery a 2 (xi X|^) is taken if Sd(k) < < 

Sc2(*^)- Third, the action mastery a 3 (xi x|^) is taken if s^ > Sc2(k). Fourth, the action 

continue sampling a 4 (x|,...,x|^) is taken if the maximum expected loss associated with this 
rule is smallest among all possible decision rules. At the final stage of sampling, n, only one 
of the three classification actions nonmastery, partial mastery, or mastery can be taken. 

In addition to the sequential cutting scores Sd(k) and Sc2(k), criteria levels td and t^2 
(0 < td < tc2 ^ 1) on the true level of functioning scale T can be identified. A student is 
considered a true nonmaster and true master if his/her true level of functioning t is smaller or 
larger than td and tc2, respectively. Furthermore, a student is considered a partial true master 
if td < t < tc2- Unlike the sequential cutting scores Sd(k) and Sc2(k), the criteria levels td 
and Iq2 do not depend on the observed item response vector (x|,...,x|^) but must be specified 
in advance by the decision-maker using methods of standard setting (e.g., Angoff, 1971; Ebel, 
1972; Nedelsky, 1954). 

Given the values of the criteria levels td and Iq 2 on T, the adaptive four-action 
mastery problem can now be stated at each stage of sampling k (1 < k < n) as choosing values 
of Sd(k) and Sc 2 (k) or continue sampling such that the maximum expected loss associated 
with the preferred rule among all possible decision rules is minimal. At the final stage of 
sampling, n, the problem reduces to choosing values of Sd(n) and Sc2(n) such that the 
maximum expected loss associated with action aj(x|,...,Xn), a2(x|,...,Xn), or a 3 (x|,...,Xn) is 
the smallest. 
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Constant Losses 

Generally speaking, a loss function evaluates for each possible action the consequences for a 
student whose true level of functioning is t. These consequences may reflect all moral, ethical, 
social, economic, psychological, etc. considerations deemed relevant. 

As earlier indicated, here the well-known threshold loss function (e.g., Ben-Shakhar & 
Beller, 1983; Chuang et al., 1981; Davis et al. 1973; Hambleton & Novick, 1973; Huynh, 
1976; Lewis & Sheehan, 1990; Novick & Lewis, 1974; Raju et al, 1991; Swaminathan et al., 
1975) is adopted as the loss structure. The choice of this function implies that the 
"seriousness" of all possible consequences of the decisions can be summarized by possibly 
different constants, one for each of the possible decision outcomes. Other loss functions have 
also been frequently used in the literature, such as linear loss (e.g., Huynh, 1980; van der 
Linden & Vos, 1996; Vos, 1990, 1991, 1995c, 1997a, 1997b, 1998a, 1998b, 1998c) and 
normal-ogive loss (e.g., Novick & Lindley, 1979; van der Linden, 1981). 

For our variable-length mastery problem, a threshold loss function can be formulated 
as a natural extension of the one for the standard fixed-length two-action mastery problem at 
each stage of sampling k (1 < k < n) as follows (see also Lewis & Sheehan, 1990): 

Table 1. Table for Threshold Loss Function at Stage k ( 1 < k < n) of Sampling 





True Level 


T<tc, 


tcl < T < tcl 


T>tc2 


Action 










ai(xi, ... 


, Xk) 


ke 


*12 + ke 


*13 + ke 


a2(xi, ... 


, Xk) 


1 1 2 + ke 


ke 


*23 + ke 


a3(xi, ... 


, Xk) 


bi + ke 


I 32 + ke 


ke 



The value e represents the costs of administering one random item. For the sake of 
simplicity, following Lewis and Sheehan (1990), these costs are assumed to be equal for each 
decision outcome as well as for each sampling occasion. Applying an admissable positive 
linear transformation (e.g.. Luce & Raiffa, 1957), and assuming the losses l| |, l22» *33 

associated with the correct decision outcomes are equal and take the smallest values, the 
threshold loss function in Table I was rescaled in such a way that \\\, l22> *33 were equal 

to zero. 
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Furthermore, the loss function associated with action ai(xi,...,xj^) must be 
nondecreasing in t, since action ai(xi,...,xj^) is most appropriate when t is small. Similarly, the 
loss function associated with action a3 (xi,...,xj^) must be nonincreasing in t due to the fact 
that action a3(xi,...,xj^) is most appropriate when t takes large values. Since it cannot be 
determined beforehand whether I21 is smaller than, equal to, or larger than I23, the form of 
the loss function associated with action a2(xi,...,xj^) is unknown. The loss associated with the 
correct partial mastery decision (i.e., l22)» however, must be the smallest. 

The loss parameters Ijj (i = 1,2,3; i ^ j) associated with the incorrect decisions have to 
be empirically assessed, for which several methods have been proposed in the literature. Most 
texts on decision theory, however, propose lottery methods (e.g., Luce & Raiffa, 1957) for 
assessing loss functions empirically. In general, the consequences of each pair of actions and 
true level of functioning are scaled in these methods by looking at the most and least preferred 
outcomes (see also Vos, 1988). 



A psychometric model is needed to specify the statistical relation between the observed 
number-correct score and student’s true level of functioning at each stage of sampling. As 
earlier remarked, here the well-known binomial model will be adopted. 

As indicated by van den Brink (1982), when tests are criterion-referenced tests by 
means of sampling from item domains, such as in our adaptive four-action mastery problem, 
the well-known binomial model is a natural choice for estimating the distribution of student’s 
number-correct score sj^ and making decisions. Hence, the binomial density function is a 
convenient choice as the psychometric model involved (see also Millman, 1972). Its 
distribution relating the observed number-correct score sj^ (0 < sj^ < k) to student's true level 
of functioning t, f(sj^ 1 1), at stage k of sampling (1 < k < n) can be written as follows: 



If each response is independent of the other, and if the examinee’s probability of a 
correct answer remains constant, the probability function of sj^, given the true level of 
^functioning t, is given by Equation 1 (Wilcox, 1981). The binomial model assumes that the 



Psychometric Model 




( 1 ) 
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test given to each student is a random sample of items drawn from a large item pool (van den 
Brink, 1982; Wilcox, 1981). Therefore, for each subject a new random sample of items must 
be drawn in practical applications of the adaptive four-action mastery problem, such as, for 
instance, in computerized adaptive instructional systems. 

Sufficient Conditions for Optimal Sequential Rules to be Monotone 

As noted earlier, the optimal sequential rules in this paper are assumed to have monotone 
forms. The restriction to monotone rules, however, is correct only if it can be proven that for 
any nonmonotone rule for the problem at hand there is a monotone rule with at least the same 
value on the criterion of optimality used (Ferguson, 1967, p.55). Using a minimax rule, the 
minimum of the maximum expected losses associated with all possible decision rules is taken 
as the criterion of optimality. 

The maximum expected loss for continuing sampling is determined by averaging the 
maximum expected loss associated with each of the possible future decision outcomes relative 
to the probability of observing those outcomes (Lewis & Sheehan, 1990). Therefore, it follows 
immediately that the conditions sufficient for setting cutting scores for the fixed-length three- 
action mastery problem, are also sufficient for the adaptive four-action mastery problem at 
each stage of sampling. Generally, conditions sufficient for setting cutting scores for the fixed- 
length multiple-decision problem are given in Ferguson (1967, p.286). 

First, f(s|^ 1 1 ) must have a monotone likelihood ratio (MLR); that is, it is required that 
for any t] > t 2 > the likelihood ratio f(s|^ I ti)/f(s|^ 1 12 ) is a nondecreasing function of S|^. MLR 
implies that the higher the observed number-correct score, the more likely it will be that the 
latent true level of functioning is high too. Second, the condition of monotonic loss must hold; 
that is, there must be an ordering of the actions such that for each pair of adjacent actions the 
loss functions possess at most one point of intersection. 

The binomial density function belongs to the monotone likelihood ratio family 
(Ferguson, 1967, chap. 5). Furthermore, it can be verified from Table 1 that for threshold loss 
the condition of monotonic loss is satisfied if at each stage of sampling k (I < k < n): 



O 



(1 j 3 +ke) - (l 23 +ke) > ( 1 1 2 +ke) - ke > ke - (I 2 1 +ke) 
(l 23 +ke) - ke > ke - ( 132 +ke) > (l 2 i+ke) - (I 3 i+ke). 



( 2 ) 
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or, equivalently, 

h3“l23^1l2^“l21 

( 3 ) 

*23 ^'*32 ^*21 'l 31 - 

Optimizing Cutting Scores for the Adaptive Four-Action Mastery Problem 

In this section, it will be shown how optimal cutting scores for the adaptive four-action 
mastery problem can be derived using the framework of minimax sequential decision theory. 
Doing so, first the minimax principle will be applied to the fixed-length three-action mastery 
problem, given an observed item response vector (x i ,...,x|^) (1 < k < n), by determining which 
of the maximum expected losses associated with the three actions a|(x|,...,x|^), a2(x|,...,X|^), 
or a3(x|,...,x0 is the smallest. 

Next, applying the minimax sequential principle, decision rules for the adaptive four- 
action mastery problem are optimized at each stage of sampling k (1 < k < n) by comparing 
this quantity with the maximum expected loss associated with action a4(xi,..,,x|^) (i.e., 
continuing sampling). 

Applying the Minimax Principle to the Fixed-Length Mastery Problem 
Given X| = X|,...,X|^ = x|^ (1 < k < n), it follows that the minimax decision rule for the fixed- 
length three-action mastery problem can be found by minimizing the maximum expected 
losses associated with the actions a| (x | ,...,X|^), a2(x | ,...,x|^), and a3(x | ,...,x|^). 

In searching for a minimax rule for the fixed-length three-action mastery problem, 
assuming the conditions of monotonicity are satisfied, we may confine ourselves to partitions 
of the range of the observed number-correct scores into three disjoint subsets A| = {s|^; s|^ < 
SclW}. A2 = {s^; Sci(k) < s^ < Sc2(k)}, and A3 = {s^; s^ > Sc2(k)} for action ai(xi,...,xk), 
a2(xi,...,x|^), and a3(x|,...,x|^) each with a conditional probability of P(A| |t), P(A2lt), and 
P(A3 1 1), respectively. It follows that the minimax decision rule can be obtained by 
minimizing the maximum expected lossess associated with the actions a|(x|,...,x|^), 
a2(xi,...,xk), anda3(xi,...,x0, or, equivalently, by minimizing the following function: 
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M(Sci(k),Sc 2 (k)) = max (L](Sci(k)),L2(scl(k),Sc2(k)).L3(sc2(k))}. (4) 



where 



Li(Sci(k))= sup l(ai(x],...,xk),t)P(Ai |t) 
iStcI 

L 2 (scl(k),Sc 2 (k))= sup l(a 2 (xi,...,x 0 ,t)P(A 2 1 1 ) 

tci<l<tc2 

L 3 (Sc 2 (k)) = sup l(a 3 (x],...,xk),t)P(A 3 1 1 ). (5) 

tatc2 

The abovementioned procedure yields the optimal cutting scores Sd(k) and Sc 2 (^) 
(i.e., the minimax cutting scores with s^iCk) < Sc 2 (k)) for a fixed-length three-action mastery 
problem with test length k. These minimax cutting scores can be obtained by computing the 
values of L|(sd(k)), L 2 (sci(k),Sc 2 (k))» and L 3 (sc 2 (^)) for all possible values of s^jCk) and 
Sc 2 (^)» with Sci(k), Sq200 = 0 ,l, 2 ,...,k, and then selecting those values s’c|(k) and s’c 2 (k) at 
which M(Sci(k),Sc 2 (k)) is the smallest. 

For our problem we are not primarily interested in determining the optimal cutting 
scores Sd(k) and Sc 2 (k)> however, but more in determining which of the three actions 
ai(x|,...,x 0 , a 2 (xi,...,x|^), or a 3 (xi,...,x|^) is optimal, given an observed item response vector 
(x|,...,x0 with number correct-score s|^ (0 < s;^ < k). In this situation it is more convenient to 
apply the following sequential procedure: First, the maximum expected losses associated with 
actions a 3 (x|,...,x|^) and a 2 (x|,...,x|^) are compared with each other. If the maximum expected 
loss associated with action a 3 is the smallest of these two quantities, then, action a 3 (xj,...,x|^) 
is taken. If the maximum expected loss associated with action a 2 (x|,...,x|^), however, is the 
smallest, then, this quantity must be compared with the maximum expected loss associated 
with action ai(x|,...,x|^). If the maximum expected loss associated with action a 2 (x|,...,x 0 is 
the smallest of these two quantities, then, action a|(x|,...,x|^) is chosen; otherwise action 
a](x|,...,x 0 is chosen. 

Applying the abovementioned procedure to a given item response vector (x|,...,x|^) 
with observed number correct-score sj^ (0 < S|^ < k), it can easily be verified from Table 1 that 
mastery (a 3 (xi,...,x|^)) is declared when the number-correct score s^ is such that 
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sup(l3,+ke)Xfv}''(l-t)“->'+ sup (l32+ke)Xf5V(l + 

‘cl<t<«c2 y=sk^^^ 



sup(ke)XfvV(l-t)‘“'' <sup(l2, +ke)X fvV(l-t)‘‘'^ + 



sup (ke)XfvV(l-t)‘‘ '' + sup(l32+ke)X^vV(l-t)‘‘■^ 

‘cl<‘«c2 y=sk'"’^^ ^^*c2 



(6) 



where y = 0,1,. .k represents all possible values the number-correct score S|^ can take after 
having observed k item responses (1 < k < n). Since the cumulative binomial density function 
is decreasing in t, it follows that the inequality in (6) can be written as: 



rk^ 






(I 3 , +ke) X ^ +(l 33 +ke) 'Z 

y=sk / y=sk^^'' 












(l32+ke)X ^ t,/(l-t„)‘‘->'+(ke)X „ 

y=Sk / y=0\y / 






k'i 



k 



(l2,+ke)X L''(l-tci)‘“''+(ke)X. 3, 

y=skV^/ y=sk'^^>^ 



tc/(l-tc2 



(ke)X L toll'd -tc,)‘‘-''+(l32 + ke)£ 

y=Sk / y=0 Vy y 



tj(l-tc2)“-^ 



(7) 



Rearranging terms, it follows that mastery is declared when the number-correct score S|^ is 
such that; 



y=sk 



G] 


^Ci) ^ "^(^23 *^^ 32 ) ^ 


^k^ 




y=«k 


\y j 



tc/0-tc2) 



c2^ -*23 



( 8 ) 



If the inequality in (8) does not hold, it can easily be verified from Table 1 that partial 
mastery (a 2 (xj,...,Xk)) is declared if it holds for number-correct score s|^ (0 < s|^ < k) that 
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sup(l2|+ke)Xfv}^(l-t)‘‘-^+ sup (ke)XftV(l-t)‘‘-^ + 

y=sic'^*^^ ‘cl<‘<‘c2 y=sic'^^^ 

sup (I23 + ke) xfv}^(l - ^ sup (ke) X ft V(’ “ + 
sup (li2+ke)XftV(l-0‘“^ + sup(l|3 + ke)XftV('"‘)‘‘‘^’ (^) 

*cl*'^*^*c2 y=sj( ^ ^ ^^*c2 y=0^ ' 

and that nonmastery (ai(xi,...,xic)) is declared otherwise. 

Rearranging terms, it follows that partial mastery is declared if the number-correct 
score is such that 



^ ^ f \C\ 

(il2“*"i2l)^ y ^ *^(^3 "“^23 ^ ^ “ ^13 “ ^23* 

y=s^\yj y=skV^>' 



and that nonmastery is declared otherwise. 



A pplying the Minimax Sequential Principle to Adaptive Mastery Testing 
The optimal sequential rule (i.e., the minimax sequential rule) at the final stage of sampling n 
is given by the minimum of the maximum expected losses associated with the actions 
a|(x|,...,Xn), a2(x|,...,Xn), and a3(x|,...,Xn), since the action a4(x|,...,Xn) (i.e., continuing 
sampling) is not available at that stage of sampling. Minimax sequential rules at the other 
stages of sampling k (i.e., 1 < k < n) are found by first computing the minimum of the 
maximum expected losses associated with the actions a|(x|,...,x|^), a2(xi,...,x|^), and 
a3(xi,...,x|^), and next comparing this quantity with the maximum expected loss associated 
with the action a4(xi,...,x|^). 

Let d|^(xi,...,x|^) denote the action a|(x|,...,x|^), a2(xi,...,x|^), or a3(x|,...,x|^) (1 < k < 
n) yielding the minimum of the maximum expected losses associated with these three actions, 
and let the maximum expected loss associated with d|^(x|,...,x|^) be denoted as V|^(x|,...,x|^). 
These notations can also be generalized to the case that no observations has been taken yet; 
that is, do(xQ) denotes the action aj(xo), a2(xQ), or a3(xQ) which yields the smallest of the 
maximum expected losses associated with these three actions, and Vo(xq) denotes the 

smallest maximum expected loss associated with do(xQ). From the foregoing it then follows 

O 







xo 
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that the minimax sequential rules for our adaptive four-action mastery problem can be found 
by using the following backward induction computational scheme: 

First, the minimax sequential rule at the final stage of sampling n is computed which is 
given by dn(xi,...,Xn); its associated maximum expected loss is given by Vn(xi,...,Xn). Next, 
the minimax sequential rule at stage (n-l) of sampling is computed by comparing 
Vn-i(xi,...,Xn.i) with the maximum expected loss associated with action a 4 (xi,...Xn.i) (i.e., 
continuing sampling). As noted before, the maximum expected loss associated with taking 
one more observation, given X| = X|,...,Xn.i = x^.i, is computed by considering all possible 
future decision outcomes at stage n (i.e., Xp = 0 or I) relative to the probability of observing 
those outcomes (i.e., backward induction). 

Let P(Xp I X] = x|,...,Xn.j = x^-i) denote the conditional distribution of X^, given 
the observed item response vector (x|,...,Xn.|), then, the maximum expected loss associated 
with taking one more observation after (n-l) observations have been taken, 
^[^n(^l» -iXn.bXn) | X j = xi,...,Xn_i = Xn_i], is computed as follows: 

^[^n(^l’***’^n-l’^n) I X| = x j,...,Xn_| = Xp.l] = 

^ n(^ 1 »---»Xfj = 0)P(Xp = 0 1 X I = X I ,...,Xp_| = Xf|_ j ) + 

Vn(xi,...,Xn = l)P(Xfj = 1 1 Xj = xi,...,Xn_l = Xp^j). (11) 

P(Xj^ I Xj = xi,...,Xj^.| = xj^.j) is also called the posterior predictive distribution of 
X|^ at stage (k-1) of sampling (1 < k < n). It will be indicated in the next section how this 
posterior predictive distribution can be computed. 

Hence, given Xj = X|,...,Xn_l = x^.j, it follows that at stage (n-l) of sampling the 
minimax sequential rule is given by: Take one more observation (i.e., action a 4 (x j,...,Xn_j)) if 
-’^n-L^n) I ^1 = Xj,..,,Xn^l = Xn-j] is smaller than Vn.|(xj,...,Xn-i), and take 
action dn_i(xj,...,Xn_i) if E[Vn(xi,...,Xn.j,Xn) | Xj = Xj,...,Xn-i = Xn.j] is larger than 
Vn-i(xj,...,Xn.i). If E[Vn(xj,...,Xn-i,Xn) | Xj =Xj,...,Xn-i = Xn-j] and Vn.i(xj,...,Xn,l) are 
equal to each other, it does not matter whether or not the decision-maker takes one more 
observation. 

Analogous to the computation at stage (n-l), the minimax sequential rule at stage (n-2) 
can be computed by comparing Vn_2(xiv,Xn-2) maximum expected loss associated 

with taking one more observation. In order to compute the maximum expected loss associated 

O 

ERIC 
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with taking one more observation at stage (n-2), first the minimum of the maximum expected 
losses associated with actions ai(x],...,Xn-i), a 2 (xi,...,Xn-i), a3(xi,...,Xn-i), and a 4 (x],...,Xn. 
]) at stage (n-1) is computed. This quantity is called the conditional minimax expected utility 
at stage (n-1), given Xj = X],...,Xn.i = Xp-i, and will be denoted as Rn-l(^h -iXn-i). It 
follows that Rn-l(xi,...,Xn_i) is equal to the minimum of Vn.i(x],...,Xn_]) and 

E[Vn(xi,...,Xn-i,Xn) I Xj =X],...,Xn^j =Xp^]]. 

At stage (n-2) of sampling, the maximum expected loss associated with taking one 
more observation after (n-2) observations, E[Rn_i(x],...,Xn_2>Xn-i) I Xj = xj,...,Xn.2 = 
Xn-2]’ can now be computed as follows: 

E[Rn- 1 (x 1 ,...,Xn-2-Xn- 1 ) I X j = x j ,...,Xn-2 = Xn-2] = 

Rn-i(x],...,Xn-l = 0 )P(Xn-i =o|Xi = X],...,Xn.2 = Xn-2) + 

Rn-l(x].--.X[].] = l)P(Xn_] = 1 IXj = X ] ,...,X[j_2 = X[j_2)- (12) 

Given Xj = x|,...,Xn_2 = Xn-2> I^e minimax sequential rule at stage (n-2) of sampling 
can now be found by comparing E[Rn.i(x],...,Xn.2.Xn-l) I Xj = xj,..., Xn _2 = Xn_2] and 
Vn-2(xi, .,Xn-2) with each other. Hence, it follows that one more observation is taken (i.e., 
action a 4 (xi,...,Xn_ 2 )) if E[Rn.](x],...,Xn_2>Xn-l) I Xj = xi,...,Xn _2 = Xn_2] 's smaller than 
Vn-2(xi. -.Xn-2)> and action dn-2(xi,...,Xn-2) is taken if E[Rn.](xi,...,Xn. 2 ,Xn.i) | Xj = 
xi,...,X[,_2 = X[,_2] is larger than V[,.2(xi. -.Xn-2)- In the case of equality between 
^n-2(xi. -.Xn-2) and E[Rn.](x],...,Xn. 2 ,Xn.i) | X] = x],...,Xn .2 = Xn_2]- ^ does not matter 
again whether or not the decision-maker takes one more observation. 

Following the same computational backward scheme as in determining the minimax 
sequential rules at stages (n-1) and (n-2), the minimax sequential rule at stage (n-3) can be 
obtained by comparing Vn. 3 (x],...,Xn. 3 ) and E[Rn. 2 (x],...,Xn. 3 ,Xn. 2 ) I X] = X],...,Xn .3 = 
^n- 3 ] with each other. The conditional minimax expected loss at stage (n-2), 
Rn-2(^b- »Xn-2)’ thereby computed inductively as the minimum of Vn-2(^1» • MXn-2) 

E[Rn-](x i,...,Xn-2»Xn-i ) I Xj = x|,...,X|^_2 ~ Xj^_2]* 

Similarly, following the same computational backward scheme, the minimax 
sequential rules at stages (n-3),.. ., 1,0 are computed. The minimax sequential rule at stage 0 
denotes the decision whether or not to take at least one observation. 
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Determination of the Least Favorable Prior 

As mentioned before, there exists a least favorable prior distribution on the true level of 
functioning such that the corresponding Bayes sequential rule is exactly the same as the 
minimax sequential rule. In computing the posterior predictive distribution P(X|^ I X| = 
X X|^_| = x|^_i), the least favorable prior on the true level of functioning is needed (1 < k 
< n). Therefore, in this section first the form of the least favorable prior will be investigated. 

Let Ip(r,s) denote the incomplete beta function with parameters r and s (r,s > 0) as 
tabulated in Pearson (1934) (see also Johnson & Kotz, 1970). It has been known for some 
time that 

=Ip(m,n-m+l). (13) 



Hence, the inequalities in (8) and (10) can be written as: 

(>31->21->32)I.ci (sk.k-Sk+1) + 023+>32)Itc2 (sk-^-Sk+l) ^ l23- (14) 

(> 1 2+>2 1 ) ltd (*k-k-sk+ 1 ) + (1 1 3-I23-1 1 2) Itc2 (sk-k-Sk+ 1 ) ^ 1 1 3 - 123- (15) 

Within the framework of Bayesian decision theory, given X| = x|,...,X|^ = X|^, it can 
be verified from Table 1 that mastery is declared for the fixed sample problem if number- 
correct score S|^ (0 < sj^ < k) is such that 

(I 3 1 +ke)P(T < tc 1 I Sk) + (l 32 +ke)P(tc i < T < tc 2 1 Sk) + (ke)P(T > tc 2 I Sk) ^ 

(I 2 1 +ke)P(T < tc 1 I Sk) + (ke)P(T > tc 2 I Sk) + d 23 +ke)P(T > tc 2 >. ( 1 6) 

Rearranging terms, it can easily be verified from (16) that mastery is declared if 

(■31-l2rl32)P(T< tci I Sk) + (l23+l32)P(’P ^ ^c2 I *k) ^ l23- (17) 
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Assuming an incomplete beta prior, it follows from an application of Bayes' theorem 
that under the assumed binomial model from (1), the posterior distribution of T will again be a 
member of the incomplete beta family (the conjugacy property, see e.g., Lehmann, 1959). In 
fact, if the incomplete beta function with parameters a and p (a, p > 0) is chosen as the prior 
distribution and student's observed number-correct score is S|^ from a test of length 
k (1 < k < n), then the posterior distribution of T is I,(a+s|^,p+k-s|^). 

Hence, assuming an incomplete beta prior, it follows from (17) that mastery is 
declared if: 

('3 1-'21->32) (a+sic.P+k-sic) + (l23+>32) Itc2 (a+sk-P+k-sk) ^ l23- (18) 

If the inequality in (18) is not satisfied, it can easily be verified from Table 1 that 
partial mastery is declared when number correct-score S|^ is such that 

(l 2 i+ke)P(T < tci I s^) + (ke)P(tci < T < tc 2 I s^) + (l 23 +ke)P(T > tc 2 1 s^) < 
(ke)P(T<tci lsk) + (li 2 +ke)P(tci <T < tc 2 I s^) + li3P(T > tc 2 I s^), (19) 

and that nonmastery is declared otherwise. Assuming an incomplete beta prior, it follows that 
partial mastery is declared if 

(1 1 2+12 1 ) Itc2 (a+Sk>P+k-Sk) + (1 1 3-I23-1 12) 1(^2 (a+s|^>P+k-s)^) ^ * 1 3“*23» (20) 

and that nonmastery is declared otherwise. 

Comparing (14) and (15) with (18) and (20), respectively, it can be seen that the least 
favorable prior for the minimax solution is given by an incomplete beta prior It(a,p) with a 
sufficiently small and p = 1. It should be noted that the parameter a > 0 can not be chosen 
equal to zero, because otherwise the prior distribution for T should be improper; that is, the 
prior does not integrate to 1 but to infinity. Hence, a must be chosen sufficiently small. 




o 
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Computation of P(Xk | Xj = = x^.i) 

To compute the maximum expected loss associated with taking one more observation at stage 
(k-1) of sampling (1 < k < n), E[Rk(xi„..,Xk.i,Xk I X\ = xi,...,Xk_i = x^.i], we need the 
posterior predictive distribution P(X|^ I X] = xi,...,X|^_i = X|^_i). From Bayes' theorem, it 
follows that: 



P(Xklx,=x,,...,Xk-i=Xk.i) = 

P(Xl =xi„..,Xk = xk)/P(Xi =xi„..,Xk.i =Xk_i). (21) 

For the binomial model as the psychometric model involved and the incomplete beta 
function as least favorable prior (with a sufficiently small), it is known (e.g., Keats & 

Lord, 1962) that the unconditional distributions of (Xi,...,X0 and (X|,...,X|^_|) are equal to: 

P(Xi =xi,...,X|^ = X|^) = 

[F(a+ 1 )r(a+sk)F( 1 +k-sk)]/[F(a)r( 1 )F(a+ 1 +k)] , (22) 

P(Xi =xi,...,Xk_i =Xk_i) = 

[T(a+1 )r(a+sk_ , )r(k-sk_ i )]/[r(a)r( 1 )r(a+k)], (23) 

where F is the usual gamma function. From (21)-(23) it then follows that the posterior 
predictive distribution of X|^, given X| = X| X|^ = x^, can be written as: 

P(Xk I Xi = xi„..,Xk.i = Xk_i) = 

[F(a+S|^)r( 1 +k“S|^)r(a+k)]/[r(a+s|^. | )F(k-s|^. | )F(a+ 1 +k)] . (24) 

Using the well-known identity F(j+1) = jF(j) and the fact that S|^ = s|^.| and sj^ = 
sk-i+1 for xj^ = 0 and 1, respectively, it finally follows from (24) that: 



P(Xk I Xi = xi„..,Xk.i = Xk.i) 



|(k-S|(.,)/(a + k) if X|( =0 
|(a + S|;.,)/(a + k) if X|; = 1. 




(25) 
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An Empirical Example 

The procedures for computing the minimax sequential decision rules were applied to a 
computerized adaptive four-action mastery test for concept-learning in medicine for freshmen. 
Concept-learning is the process in which subjects learn to categorize objects, processes or 
events, for instance, formation of diagnostic skills in medicine or psychology (see Tennyson 
and Cocchiarella, 1986, for a complete review of the theory of concept-learning). 

In order to comply with the assumption of the binomial density as psychometric model 
involved, it was necessary to draw for each of the students a new random sample of items 
from a large pool of items. 

The instructors of the program considered students as 'true masters' if they had 
mastered at least 60% of the total number of items covering the subject matter of that concept. 
Therefore, t ^2 was fixed at 0.6. Furthermore, students were considered as 'true nonmasters' if 
they had mastered less than 50% of the total number of items covering the subject matter of 
the present concept. Therefore, Iq\ was fixed at 0.5. 

Finally, the constant cost for administering one random item was assumed to be rather 
small; hence, the value of e was set equal to 0.05. 

Results for the Minimax Sequential Rules 

Using the lottery method discussed in Vos (1988), assuming equal losses for the correct 
decisions In, l 22 > and I 33 and taking into account the requirements I 31 > I 32 and 1 13 > 1 12 , 
the losses from Table 1 were empirically assessed by the instructors of the program; the 
results are presented in Table 2. 



Table 2. Threshold Loss Table at Stage k (1 < k < n) of Sampling for Empircal Example 





True Level 


T<tci 


tcl <T<tc| 


T>tc2 


Action 










ai(xi, ... 


, Xk) 


ke 


2 + ke 


4 + ke 


a2(xi, ... 


, Xk) 


1 +ke 


ke 


2 + ke 


a3(xi, ... 


, Xk) 


3 + ke 


1 +ke 


ke 



it 




As noted before, the sign of [l21~*23l could not be determined beforehand. In Table 2, 
is assumed that I 23 > I 21 (i c., I 23 = 2 and I 21 = I). In other words, for this specific 
ipirical example it is assumed that the loss associated with taking action a 2 is twice as large 
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for a student whose true level of functioning exceeds than for a student whose true level of 
functioning is below td . 

Using the numerical values for the loss parameters Ijj (ij = 1,2,3) of Table 2, for a 
maximum of 30 items (i.e., n = 30), the appropriate action (i.e., nonmastery, partial mastery, 
mastery, or continue sampling) is depicted in Table 3 at each stage of sampling k (0 < k < n) 
for different number-correct score sj^ (0 < sj^ < k) as a closed interval. 



Table 3. Appropriate Action Calculated by Stage of Sampling and Number-Correct 



Stage of sampling 




Appropriate Action by Number-Correct 






Nonmastery 


Continue 


Partial 

Mastery 


Continue 


Mastery 


0 




0 








1 


0 


1 








2 


0 


[1,2] 








3 


0 


[1,3] 








4 


0 


[1,3] 






4 


5 


[0,1] 


[2,4] 






5 


6 


[0,1] 


[2,5] 






6 


7 


[0,2] 


[3,5] 






[6,7] 


8 


[0,2] 


[3,6] 






[7,8] 


9 


[0,3] 


[4,6] 






[7,9] 


10 


[0,3] 


[4,7] 






[8,10] 


11 


[0,4] 


[5,7] 






[8,11] 


12 


[0,4] 


[5,8] 






[9,12] 


13 


[0,5] 


[6,9] 






[10,13] 


14 


[0,5] 


[6,9] 






[10,14] 


15 


[0,6] 


[7,10] 






[11,15] 


16 


[0,6] 


[7,10] 






[11,16] 


17 


[0,7] 


[8,11] 






[12,17] 


18 


[0,7] 


[8,11] 






[12,18] 


19 


[0,8] 


9 


10 


[11,12] 


[13,19] 


20 


[0,8] 


[9,10] 


11 


12 


[13,20] 


21 


[0,9] 


10 


11 


[12,13] 


[14,21] 


22 


[0,9] 


[10,11] 


12 


[13,14] 


[15,22] 


23 


[0,10] 


11 


12 


[13,14] 


[15,23] 


24 


[0,10] 


[11,12] 


13 


[14,15] 


[16,24] 


25 


[0,11] 


12 


[13,14] 


15 


[16,25] 


26 


[0,11] 


[12,13] 


14 


[15,16] 


[17,26] 


27 


[0,12] 


13 


[14,15] 


16 


[17,27] 


28 


[0,12] 


13 


[14,16] 


17 


[18,28] 


29 


[0,13] 


14 


[15,16] 


17 


[18,29] 


30 


[0,14] 




[15,17] 




[18,30] 
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Table 3 was constructed by using the following backward induction computational 
scheme. First, the appropriate action at each stage of sampling k (0 < k < n) for the fixed- 

length three-action mastery problem (i.e., d|^(x| x|^)) was determined by examining if the 

inequalities in ( 8 ) and (10) were satisfied. More specifically, mastery was declared for those 
values of s^ (0 < sj^ < k) for which the inequality in ( 8 ) was satisfied; partial mastery was 
declared for those values of s|^ for which the inequality in ( 8 ) was not satisfied but the 
inequality in ( 10 ) did hold; and mastery was declared for those values of s|^ for which both the 
inequalities in ( 8 ) and (10) were not satisfied. Note that it can be inferred from Table 2 that 
Scl(30) and s^2(30) are equal to 14 and 18, respectively. 

Next, the appropriate action nonmastery, partial mastery, or mastery and its associated 
maximum expected loss were computed at stage 29 of sampling for S 29 = 0,...,29 (i.e., 
d 29 (xi,...,X 29 ) and V 29 (x|,...,X 29 )). Using (11), (25), and the maximum expected losses 
calculated at the final stage of sampling, the maximum expected loss associated with taking 
one more observation at stage 29 of sampling was computed for S 29 = 0,...,29 (i.e., 
E[^30(^1» •’^29’^30) I ^1 = X|,...,X 29 = X 29 ]). Comparing these values with 
V29 (xi„..,X 29), the appropriate action nonmastery, partial mastery, mastery, or continue 
sampling was determined at stage 29 of sampling. 

In order to compute the appropriate action nonmastery, partial mastery, mastery, or 
continue sampling at stage 28 of sampling, first the conditional minimax expected loss at 
stage 29 of sampling was computed (i.e., R29(xi,...,X29)) by taking the minimum of 
V 29 (xi,-.,X 29 ) and E[V 3 o(xi„..,X 29 ,X 3 o) I Xj = X],...,X 29 = X 29 ] for S 29 = 0,...,29. Next, 
using (12) and (25), the maximum expected losses associated with taking one more 
observation at stage 28 of sampling, E[R 29 (x|,...,X 28 ,X 29 ) I X| = X|,...,X28 = X28l were 
computed for S28 = 0,...,28. Comparing these values with V28(x|,...,X28)> the appropriate 
action nonmastery, partial mastery, mastery, or continue sampling was determined at stage 28 
of sampling. 

Similarly, the appropriate action at stage 27 until stage 0 of sampling was determined. 
The conditional minimax expected loss at stage 28 of sampling, R28(^l’ * ’^28)» was thereby 
computed inductively as the minimum of V28(xi,...,X28) and E[R 29 (x|,...,X 28 »X 29 ) I X] = 
xi,...,X28 = X28] for S28 = 0,...,28. 
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Using the recurrent relation 











^ k ^ 




— 




+ 





, in combination with 
















1, for computing the binomial coefficients in (8) and (10), a computer program called 
MINIMAX was developed to determine the appropriate action at each stage of sampling. 
Furthermore, the parameter a in (25) of the incomplete beta prior was set equal to 10"^. A 
copy of the program MINIMAX is available from the author upon request. 

As can be seen from Table 3, the decision-maker takes at least one observation. Table 
3 also shows that at stage k of sampling, a student whose percentage number-correct (i.e., 
s|^/k) is in the region of Iq\ or t^2 is hard to be classified as a nonmaster, partial master, or 
master. For such students longer tests are needed, since continue sampling is the appropriate 
action in this situation. Shorter tests can be provided, however, for students whose percentage 
number-correct is not in the region of t^i or tc2- The main advantage of variable-length 
mastery testing, as stated already in the Introduction, is demonstrated here clearly; that is, the 
test is adapted to the actual level of functioning. 

Furthermore, Table 3 shows that continue sampling decisions are taken for the first 
time after 18 items have been administered in situations where neither clearly partial mastery 
nor mastery can be declared. Continue sampling decisions are taken at each stage of sampling, 
however, in situations where neither clearly nonmastery nor partial mastery can be declared. 

This finding might be accounted for that the losses associated with taking false 
nonmastery decisions are twice as large as the losses associated with taking false partial 
mastery decisions (i.e., 2 and 4 relative to 1 and 2), whereas the differences between the losses 
associated with taking false partial mastery and mastery decisions (i.e., 1 and 2 relative to 3 
and 1) are not that large. Consequently, it seems better to continue sampling if students neither 
have clearly attained a certain level of nonmastery nor partial mastery in order to avoid 
relatively large expected losses associated with taking false decisions. 

Finally, it can be inferred from Table 3 that with increasing number of items being 
administered, less continue sampling decisions are taken; that is, the probability of a 
classification decision increases. 

Since at least one observation is taken, the minimax sequential procedure starts with 
administering one ramdomly selected item and stops after a classification decision (i.e., 
declaring nonmastery, partial mastery, or mastery). Hence, the minimax sequential procedure 
proceeds only after the continue sampling decision. Then, it can easily be inferred from Table 
that the minimax sequential decision rule, in case of stopping sampling after a classification 
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decision, can be depicted in Table 4 at each stage of sampling k (1 < k < 30) for different 
number-correct score sj^ (0 < sj^ < k) as follows: 



Table 4. Minimax Sequential Rule in Case of Stopping Sampling after Classification 



Stage of sampling 




Minimax Sequential Rule by Number-Correct 






Nonmastery 


Continue 


Partial 

Mastery 


Continue 


Mastery 


1 


0 


1 








2 




[1,2] 








3 




[1,3] 








4 




[1,3] 






4 


5 


1 


[2,4] 








6 




[2,5] 








7 


2 


[3,5] 






6 


8 




[3,6] 








9 


3 


[4,6] 






7 


10 




[4,7] 








11 


4 


[5,7] 






8 


12 




[5,8] 








13 


5 


[6,9] 








14 




[6,9] 






10 


15 


6 


[7,10] 








16 




[7,10] 






11 


17 


7 


[8,11] 








18 




[8,11] 






12 


19 


8 


9 


10 


[11,12] 




20 




[9,10] 


11 


12 


13 


21 


9 


10 


11 


[12,13] 




22 




[10,11] 


12 


[13,14] 




23 


10 


11 


12 


[13,14] 


15 


24 




[11,12] 


13 


[14,15] 




25 


11 


12 


[13,14] 


15 


16 


26 




[12,13] 




[15,16] 




27 


12 


13 


[14,15] 


16 


17 


28 




13 


14 or 16 


17 




29 


13 


14or 17 






18 


30 


14 




15 or 17 




18 



Note that not all possible number-correct scores sj^ are necessarily present at each 
stage of sampling k, because it is assumed in Table 4 that the minimax sequential rule stops 
after classifying a student as a nonmaster, partial master, or master. For instance, the number- 
correct score S5 can only take the values 1 until 4, and thus, not the values 0 and 5. This is 
because nonmastery and mastery was already declared for S| =0 and S4 = 4, respectively. 
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implying the minimax sequential rule stops for sj = 0 or S 4 = 4 . It follows that S 2 , S 3 , S 4 , and 
S 5 can only take the values [1,2], [1,3], [1,4], and [1,4], respectively. 

As can be seen from Table 4, at the final stage of sampling (i.e., stage 30) the number- 
correct score S 30 can only take the values 14, 15, 17, or 18. In other words, values larger than 
18 and smaller than 14 can not occur for S 3 Q, whereas the value of 16 can not occur as well. 
This is because for all possible values of S 30 equal to or smaller than 14, students have already 
been clearly classified as a master, partial master, or nonmaster before the final stage of 
sampling has been reached. For the possible value of S 30 = 16, students could possibly have 
been classified already at stage 25 or 28 as a master or partial master, respectively. Possible 
values of S 30 larger than 18 can not occur, since students would have clearly attained already 
a certain level of mastery, partial mastery, or nonmastery for values of sj^ (1 < k < 30) equal to 
or smaller than 18. 



Optimal rules for the adaptive four-action mastery problem (nonmastery, partial mastery, 
mastery, and continuing sampling) were derived using the framework of minimax sequential 
decision theory. The binomial distribution was assumed for the psychometric model involved, 
whereas threshold loss was adopted for the loss function. The least favorable prior, needed for 
computing the maximum expected loss associated with continuing sampling, turned out to be 
the incomplete beta prior with parameter a sufficiently small and parameter P equal to 1 . The 
minimax sequential rules were demonstrated by an empirical example for concept learning in 
medicine, with a maximum test length of 30. 

The results indicated that the chances of being classified as a nonmaster, partial 
master, or master increased if the number of items administered increased. This result was in 
accordance with our expectations. Furthermore, the minimax sequential decision rules in case 
of stopping sampling after a classification decision were computed at each stage of sampling 
for different number-correct score. It turned out that, for instance, at the final stage of 
sampling (i.e., after 30 items have been administered) the number-correct score could only 
take the values of 14, 15, 17, or 18. 

It is important to notice that, as pointed out by Veldhuijzen (1982) and Huynh (1980), 
the minimax principle is very attractive when the only information is the student's observed 
""mber-correct score; that is, no group data of 'comparable' students or prior information 
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about the individual student is available. This situation is often the case in individualized 
instructional programs. If group data of ’comparable' students or prior information about the 
individual student is available, however, it is better to use this information in the minimax 
sequential decision procedure regarding declaring mastery, partial mastery, or nonmastery. 
Hence, in this situation it is better to use Bayesian instead of minimax sequential decision 
theory. Even if information in the form of group data of 'comparable' students or prior 
information about the individual student is available, it is sometimes too difficult a job to 
accomplish to express this information into a prior distribution. In these circumstances, the 
minimax sequential procedure might also be more appropriate. 

Several procedures have been proposed which are simple variants of the minimax 
strategy (e.g., Coombs, Dawes, & Tversky, 1970), and may also be applied to the adaptive 
four-action mastery problem. The first is the minimin (complete optimism) strategy, where 
optimal rules are obtained by minimizing the minimum expected losses associated with all 
possible decision rules. This strategy is optimal if the best that could happen always happens. 
As pointed out by Veldhuijzen (1982), the minimin strategy seems rather useless in the 
context of individualized instructional programs. 

The second is the pessismism-optimism (Hurwicz) strategy. This strategy is a 
combination of the minimax and minimin strategies, and decisions are taken on the basis of 
the smallest and largest expected losses associated with each possible decision outcome. 

The third is the minimax-regret (Savage) strategy. This strategy is similar to the 
minimax strategy since the focus is on the worst possible decision outcome, but 'worst' is here 
defined by maximal regret; that is, the difference between the maximal expected loss that was 
actually obtained, and the maximal expected loss among all possible decision outcomes. 

Two final notes are appropriate. First, it might be assumed that guessing has to be 
taken into account. Huynh (1980) has developed a model with corrections for guessing within 
a minimax decision-theoretic framework (see also van den Brink & Koele, 1980) for the 
fixed-length two-action mastery problem (i.e., mastery/nonmastery) in case of a linear loss 
function. This framework of correction for guessing (i.e., know ledge-or-random -guessing) can 
be used to derive minimax sequential rules for the adaptive four-action mastery problem. 

Second, the minimax sequential decision procedures presented in this paper were 
applied to the problem of deciding on nonmastery, partial mastery, nonmastery, or continuing 
sampling in the context of computerized adaptive mastery testing. It should be emphasized, 
‘'owever, that the decision-making procedures advocated here have a larger scope. For 
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instance, the adaptive four-action mastery problem may be important in clinical settings where 
therapies are followed by mastery tests, which decide on whether or not patients are dismissed 
from the therapies. 
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